Are we forgetting how the white Dutch stole their farms from the indigenous people
The reality is a little less clear-cut than one might imagine. Southern Africa's indigenous peoples were of a group of non-Bantu people, referred to collectively as the "Khoisan". This included the hunter-gatherer San people (formerly called "Bushmen") as well as the pastoralist Khoikhoi (formerly known as "Hottentots").
At the time the Dutch established their 17th Century outpost in what is now Capetown, they found the district already settled by the Khoikhoi. These peoples had previously experienced displacement from the north by various migrations of the Bantu, and now faced similar displacement from Dutch settlement. Later, during a period known as the "Mfecane" (1815 - 1840) widescale warfare between various African (i.e. Bantu) tribes left large areas depopulated; and therefore ripe for settlement by others - including Trek-Boers from the Cape.
Regarding the whole of Southern Africa, the matter of who did what to who is just a little bit more involved than merely the good guys wearing white hats / black skins and the bad guys wearing black hats / white skins. (For a starter, the "indigenous" people weren't black; but were more Asian in appearance).